Papers with regularization methods
Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has revealed that machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. |
| Approach: | They propose a metric to detect spurious tokens and a family of regularization methods to mitigate spurious correlations in text classification. |
| Outcome: | The proposed method prevents spurious clusters and significantly improves the robustness of classifiers without auxiliary data. |
A Regularization-based Framework for Bilingual Grammar Induction (D19-1)
Copied to clipboard
| Challenge: | Existing multilingual grammar induction methods require external resources such as parallel corpora, word alignments or linguistic phylogenetic trees. |
| Approach: | They propose a framework in which the learning process of the grammar model of one language is influenced by knowledge from the model of another language. |
| Outcome: | The proposed method outperforms baselines on transfer grammar induction and bilingual grammar inducing on multiple languages. |
Exploring Robust Overfitting for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent literature has revealed their vulnerability to crafted adversarial examples on a wide range of NLP tasks. |
| Approach: | They propose to combine regularization methods with adversarial training to mitigate robust overfitting for pre-trained language models by scaling the model's loss. |
| Outcome: | The proposed methods mitigate robust overfitting upon three top adversarial training methods and further promote adversarially robustness. |
Sequence Length is a Domain: Length-based Overfitting in Transformer Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current Transformer-based sequence-to-sequence architectures can suffer from overfitting during training. |
| Approach: | They propose to use Transformer-based sequence-to-sequence architectures to overcome overfitting problems when generating very long sequences. |
| Outcome: | The proposed model performs worse on very long sequences than previous approaches on string editing and translation tasks when faced with sequences of length diverging from the length distribution in training data. |
Revisiting Pathologies of Neural Models under Input Reduction (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that modern neural models tend to be miscalibrated. |
| Approach: | They examine why models produce high-confidence predictions on inputs that appear nonsensical to humans . previous work suggested that models fail to assign low probabilities due to model overconfidence . |
| Outcome: | The proposed methods can be extended to reduce the number of examples but with the cost of miscalibration. |
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |
IR2: Information Regularization for Information Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Effective information retrieval (IR) in settings with limited training data remains a challenging task. |
| Approach: | They propose a technique for reducing overfitting during synthetic data generation . they use DORIS-MAE, ArguAna, and WhatsThatBook as examples . |
| Outcome: | The proposed technique outperforms previous methods and reduces cost by 50% on three recent IR tasks characterized by complex queries. |
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Sparsely gated Mixture of Experts (MoE) models are a compute-efficient method to scale model capacity for multilingual machine translation tasks. |
| Approach: | They propose a regularization strategy that prevents over-fitting of MoE models on low-resource tasks and conditional MoE Routing and curriculum learning methods that prevent over- fitting. |
| Outcome: | The proposed methods improve the performance of MoE models on low-resource tasks without adversely affecting high-res tasks. |
Lifelong Knowledge Editing requires Better Regularization (2025.findings-emnlp)
Copied to clipboard
Akshat Gupta, Phudish Prateepamornkul, Maochuan Lu, Ahmed Alaa, Thomas Hartvigsen, Gopala Anumanchipalli
| Challenge: | Knowledge editing is a promising way to improve factuality in large language models, but recent studies have shown significant model degradation during sequential editing. |
| Approach: | They formalize locate-then-edit methods as a two-step fine-tuning process . they show that model degradation occurs due to over-optimization of internal activations . |
| Outcome: | The proposed methods reduce time and improve factuality by 42-61%. |